Load Balance and Communication Tradeo s in Parallel

In block-partitioned parallel matrix factorization algorithms, where the matrix is distributed over a logical torus processor grid with an rs block-cyclic matrix distribution, the greatest scope for optimization exists in the formation of (block) panels. Let ! be the panel width, with ! m being an optimal value based on the characteristics a single processor's memory hierarchy. To date, two well-known techniques to do this are known as storage blocking, where the ! m ! = r = s, and algorithmic blocking (also known as`dis-tributed panels'), where ! ! m ; r = s 1. These represent strategies at opposite ends of a load balance and communication cost tradeoo In this paper, we present two new techniques for the panel formation, called pipelining with lookahead, and panel scattering. The former requires communication to be uni-directional across a processor dimension, and thus can normally only be applied to the column panel. It can be characterized by ! m ! = s, with communication and computation overlapped across processor columns, at the cost of some pipeline startup time. The latter uses ! m ! = r = s, but involves scattering the panel across its longest dimension across all processors, to be collected and broadcasted when the panel formation is complete. While it achieves perfect load balance, this method can double the communication volume. Implementation issues for these methods will be discussed. For a given target architecture, the optimum method (or combination of methods) depends on the communication to computation performance of that architecture.